System and method for displaying video surveillance fields of view limitations

ABSTRACT

A system and method are provided for displaying video surveillance fields of view limitations. The system and method perform video surveillance of a given area and then geo-locate any obstacles within the area, including measuring their overall size and shape. The system and method map the size, location and shape of the objects into a database and then identify where there are video surveillance coverage gaps from each vantage point where video surveillance is being performed on the area. The system and method then determine where there are overlapping locations of blocked video surveillance (i.e., locations that are “invisible” to video surveillance). The system and method create a simulated image of the area from an orientation above the area which indentifies locations within the area that may not be seen by video surveillance from any of the vantage points where video surveillance is being performed.

BACKGROUND

One of the drawbacks with using video surveillance to monitor a locationis that it can be difficult to determine where there are coverage gapsin the surveillance. This difficulty is exacerbated when reconcilingsurveillance coverage from multiple viewpoints (i.e., when several videocameras are used to cover an area from multiple locations).

The video cameras in a typical security system are usually placed suchthat all of the scenes which are viewed by the cameras overlap to someextent. However, there are often areas where one or more obstacles blocka portion of the field of view of one camera and the remaining camerasare unable to provide adequate surveillance of the blocked area. Thesegaps in the video surveillance may not be readily apparent when cameradata is viewed by security personnel.

One method that is used to minimize the size and number of blocked videocoverage areas is to place surveillance cameras at optimal locationssuch that the effect of obstacles is minimized. The placement of camerasin these desired positions can often be problematic because there may beno infrastructure or supporting structures that exist at these locationsmaking it difficult and/or expensive to adequately mount the videocameras. In addition, even if special arrangements are made to placecameras at these locations, there are typically unforeseen areas ofblocked coverage.

Another of the current methods that is used to minimize the size andnumber of blocked video coverage areas is to place multiple cameras inan area and use rotating field of views for each of the cameras. One ofthe shortcomings associated with using rotating field of views for eachof the cameras is that events in the field of view of the camera cantranspire when the camera is not pointing where the events occur.Security personal monitoring multiple screens, and particularly screenswith rotating fields of view, frequently fail to detect activity onthose screens. In addition, even when rotating field of views are usedfor each of the cameras, there are typically unforeseen areas of blockedcoverage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating a method for displaying videosurveillance fields of view limitations according to an exampleembodiment.

FIG. 2 illustrates a system for displaying video surveillance fields ofview limitations according to an example embodiment.

FIG. 3 illustrates an example simulated image that may be generated bythe system and method for the area that is shown in FIG. 2.

FIG. 4 shows examples of a lidar and camera combination.

FIG. 5 illustrates the system of FIG. 2 where objects within thesurveillance area have been moved.

FIG. 6 illustrates an example simulated image that may be generated bythe system and method for the area that is shown in FIG. 5.

FIG. 7 is a block diagram of a typical computer system used to implementportions of methods according to an example embodiment.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments which may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, and it is to be understood thatother embodiments may be utilized and that structural, logical andelectrical changes may be made without departing from the scope of thepresent invention. The following description of example embodiments is,therefore, not to be taken in a limited sense, and the scope of thepresent invention is defined by the appended claims.

The functions or algorithms described herein may be implemented insoftware or a combination of software, hardware and human implementedprocedures in one embodiment. The software may consist of computerexecutable instructions stored on computer readable media such as memoryor other type of storage devices. Further, such functions correspond tomodules, which are software, hardware, firmware or any combinationthereof. Multiple functions may be performed in one or more modules asdesired, and the embodiments described are merely examples. The softwaremay be executed on a digital signal processor, ASIC, microprocessor, orother type of processor operating on a computer system, such as apersonal computer, server or other computer system.

A system and method are provided for displaying video surveillancefields of view limitations. In some embodiments, the system and methodperform video surveillance of a given area and then geo-locate anyobstacles within the area, including measuring their overall size andshape. The system and method further map the size, location and shape ofthe objects into a database and then identify where there are videosurveillance coverage gaps from each vantage point where videosurveillance is being performed on an area.

The system and method then determine where there are overlappinglocations of blocked video surveillance (i.e., locations that are“invisible” to video surveillance). The system and method create asimulated image of the area from an orientation above the area whichindentifies locations within the area that may not be seen by videosurveillance from any of the vantage points where video surveillance isbeing performed. The simulated image provides a vivid display of thoselocations within the area that are vulnerable to inadequate videomonitoring.

FIG. 1 is a flowchart illustrating a method 100 for addressing videosurveillance field of view limitations according to an exampleembodiment. The method 100 comprises activity 110 which includes loadinga first set of data relating to the size and distance of objects in anarea from a first vantage point into a database; activity 120 whichincludes loading a second set data relating to the size and distance ofthe objects in the area from a second vantage point into the database;activity 130 which includes loading a global position of the firstvantage point and the second vantage point into the database; activity140 which includes determining a global position of the objects in thearea based on information in the database; and activity 150 whichincludes using information in the database to create a simulated imageof the area from an orientation above the area which indentifieslocations within the area that may not be seen by video surveillancefrom the first the vantage point and the second vantage point (see,e.g., FIG. 3).

In some example embodiments, the activity 110 of loading the first setof data into the database and the activity 120 of loading the second setof data into the database may include using a lidar (i.e., LightDetection and Ranging or Laser Imaging Detection and Ranging (system),or Laser Identification Detection and Ranging or Laser InducedDifferential Absorption Radar) to obtain the first set of data and thesecond set of data. As an example, the activity 110 of loading the firstset of data into the database may include using a first lidar to obtainthe first set of data and the activity 120 of loading the second set ofdata into the database may include using a second lidar to obtain thesecond set of data. In addition, using the first lidar to obtain thefirst set of data includes may include positioning the first lidar atthe first vantage point and using the second lidar to obtain the secondset of data may include positioning the second lidar at the secondvantage point. It should be noted that in some embodiments, using thefirst lidar to obtain the first set of data may be done simultaneouslywith using the second lidar to obtain the second set of data.

The method 100 may further include the activity 124 which includesloading a first video image of the area from the first vantage pointinto the database and the activity 126 which includes loading a secondvideo image of the area from the second vantage point into the database.When these types of first and second video images are loaded into thedatabase, the activity 150 of using information in the database tocreate a simulated image of the area from an orientation above the areawhich indentifies locations within the area that may not be seen byvideo surveillance from the first the vantage point and the secondvantage point may include indentifying locations within the area thatmay not be seen by the first video image and the second video image.

In some example embodiments, the activity 124 of loading a first videoimage of the area may include recording the first video image with afirst camera, and the activity of loading a second video image of thearea may include recording the second video image with a second camera.It should be noted that recording the first video image may be donesimultaneously with recording the second video image. In addition,recording the first video image with the first camera may includepositioning the first camera at the first vantage point and recordingthe second video image with the second camera may include positioningthe second camera at the second vantage point.

In some example embodiments, activity 130 which includes loading aglobal position of the first vantage point and the second vantage pointinto the database may further include determining the global position ofthe first vantage point and determining the global position of thesecond vantage point. As an example, determining the global position ofthe first vantage point may be done simultaneously with determining theglobal position of the second vantage point by using a globalpositioning system that includes components which are located at thefirst vantage point and the second vantage point.

In some example embodiments, the activity 140 of determining a globalposition of the objects in the area based on information in the databasemay include monitoring movement of the objects within the area. Thedetermination may be based on knowing the global position of the firstvantage point and the second vantage point as well as continuouslymonitoring the locations of the objects in the area relative to thefirst vantage point and the second vantage point.

One example of where this may be useful is for areas such as shippingports where stacks of shipping containers are constantly moving in andout of a port (i.e., a surveillance area). As the containers stack up orare moved, there will be changing gaps in the coverage of the videosurveillance system.

FIG. 2 illustrates a video surveillance system 10 according to anexample embodiment. The video surveillance system 10 includes a firstlidar 12 that is located at a first vantage point X and a second lidar14 that is located at a second vantage point Y.

The video surveillance system 10 further includes a global positioningsystem 20 that detects the global position of the first lidar 12 and thesecond lidar 14. The global positioning system 20 and the first andsecond lidars 12, 14 are used to globally locate objects O1, O2, O3, O4,O5 within an area A that is being monitored by video surveillance anddetermines the size and shape of the objects O1, O2, O3, O4, O5.

The video surveillance system 10 further includes a processor 30 thatreceives data from the first lidar 12, the second lidar 14 and theglobal positioning system 20. Based on the received data, the processor30 creates a simulated image of the area from an orientation above thearea A which indentifies locations within the area that may not be seenby video surveillance from the first vantage point and the secondvantage point. FIG. 3 illustrates an example simulated image that may begenerated for the area A and objects O1, O2, O3, O4, O5 using the lidars12, 14 that are shown in FIG. 2.

The video surveillance system 10 may further include a first camera 16that is located at the first vantage point X and a second camera 18 thatis located at the second vantage point Y. The size, shape and locationof the objects O1, O2, O3, O4, O5 within the area A may be correlatedwith video images that are taken from the first and second video cameras16, 18. In addition, the global positioning system 20 may be mounted onthe first camera 16 and the global positioning system 20 may be mountedon the second camera 18.

FIG. 4 shows examples of a lidar and camera combination. In theillustrated example embodiments, the first camera 16 is mounted to thefirst lidar 12 and the second camera 18 is mounted to the second lidar14 such that the global positioning system 20 is mounted to both thefirst camera 16 and the first lidar 12 and the global positioning system20 is mounted to both the second camera 18 and the second lidar 14.

When the first and second lidars 12, 14 are mounted on the first andsecond cameras 16, 18 (or vice versa), the surveillance system 10 may beable to continuously update the data to display those areas that areblocked from video surveillance by the first and second cameras 12, 14from an orientation above the area. In some example embodiments, thefirst video camera 16 and the second video camera 18 simultaneously senddata to the processor 30 and/or the first lidar 12 and the second lidar14 simultaneously send data to the processor 30. In addition, the globalpositioning system 20 may simultaneously send data to the processor 30along with the first and second lidar 12, 14 and/or the first and secondvideo cameras 16, 18.

As discussed above, one example of where this may be useful is for areassuch as shipping ports where stacks of shipping containers areconstantly moving in and out of a port (i.e., a surveillance area). Asthe containers stack up or are moved, there will be changing gaps in thevideo surveillance.

FIG. 5 shows an example of where some of the objects O3, O4, O5 shown inFIG. 2 have moved within the area A relative to the first and secondlidars 12, 14 and the first and second cameras 16, 18. FIG. 6illustrates an example simulated image that may be generated for thearea A and objects O1, O2, O3, O4, O5 using the lidars 12, 14 that areshown in FIG. 5.

Although not explicitly shown in the FIGS., the first and second lidars12, 14 and the first and second cameras 16, 18 are able to monitor whena portion of an object may be moved within, or removed from, the area A.As an example, the system 10 is able to monitor when one or morecontainers in a stack of containers is removed from the rest of thestack of containers.

It should be noted that embodiments are contemplated where only a singlelidar and/or camera combination is used to supply data to the processor30 relating to the size and distance of objects in the area A from thefirst vantage point X and then subsequently supply data relating to thesize and distance of objects in the area A from the second vantage pointY. In addition, a single component in the global positioning system 20may be used to supply the global position of the first and secondvantage points X, Y to the processor 30.

Embodiments are also contemplated where multiple lidars and/or camerasare used to supply data to the processor 30 relating to the size anddistance of objects in the area A from multiple vantage points. Inaddition, multiple components in the global positioning system 20 may beused to supply the global positions of the multiple vantage points tothe processor 30.

In some embodiments, a computer system may form part of the system 10. Ablock diagram of an example computer system that executes programmingfor performing some of the methods described above is shown in FIG. 7.

A general computing device in the form of a computer 710, includes aprocessing unit 7O2 (e.g., processor 30), memory 7O4, removable storage712, and non-removable storage 714. Memory 7O4 may include volatilememory 706 and non-volatile memory 708. Computer 710 may include—or haveaccess to a computing environment that includes—a variety ofcomputer-readable media, such as volatile memory 706 and non-volatilememory 708, removable storage 712 and non-removable storage 714. Itshould be noted that the databases referred to above for crating thesynthetic image may be part of any of the processing unit 7O2 (e.g.,processor 30), memory 7O4, volatile memory 706, non-volatile memory 708,removable storage 712, and non-removable storage 714.

Computer storage includes random access memory (RAM), read only memory(ROM), erasable programmable read-only memory (EPROM) & electricallyerasable programmable read-only memory (EEPROM), flash memory or othermemory technologies, compact disc read-only memory (CD ROM), DigitalVersatile Disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium capable of storing computer-readable instructions,as well as data, including video frames.

Computer 710 may include or have access to a computing environment thatincludes input 716, output 718, and a communication connection 720. Insome example embodiments, the input 716 may allow a user to select thedisplayed size and level of detail within the simulated image. Inaddition, the output 718 may include a display that illustrates theoverhead simulated image generated by the processor 30.

The computer may operate in a networked environment using acommunication connection to connect to one or more remote computers. Theremote computer may include a personal computer (PC), server, router,network PC, a peer device or other common network node, or the like. Thecommunication connection may include a Local Area Network (LAN), a WideArea Network (WAN) or other networks. Computer-readable instructionsstored on a computer-readable medium are executable by the processingunit 7O2 of the computer 710. A hard drive, CD-ROM, and RAM are someexamples of articles including a computer-readable medium. The Abstractof the Disclosure is provided to comply with 37 C.F.R. §1.72(b) with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. The above description and figures illustrateembodiments of the invention to enable those skilled in the art topractice the embodiments of the invention. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separate embodiment.

1. A method of displaying video surveillance fields of view limitations,the method comprising: loading a first set of data relating to the sizeand distance of objects in an area from a first vantage point into adatabase; loading a second set data relating to the size and distance ofthe objects in the area from a second vantage point into the database;loading a global position of the first vantage point and the secondvantage point into the database; determining a global position of theobjects in the area based on information in the database; and usinginformation in the database to create a simulated image of the area froman orientation above the area which indentifies locations within thearea that may not be seen by video surveillance from the first thevantage point and the second vantage point.
 2. The method of claim 1further comprising: loading a first video image of the area from thefirst vantage point into the database; loading a second video image ofthe area from the second vantage point into the database; and whereinusing information in the database to create a simulated image of thearea from an orientation above the area which indentifies locationswithin the area that may not be seen by video surveillance from thefirst the vantage point and the second vantage point includesindentifying locations within the area that may not be seen by the firstvideo image and the second video image.
 3. The method of claim 1 whereinloading a first video image of the area includes recording the firstvideo image with a first camera, and wherein loading a second videoimage of the area includes recording the second video image with asecond camera.
 4. The method of claim 3 wherein recording the firstvideo image is done simultaneously with recording the second videoimage.
 5. The method of claim 4 wherein recording the first video imagewith the first camera includes positioning the first camera at the firstvantage point, and wherein recording the second video image with thesecond camera includes positioning the second camera at the secondvantage point.
 6. The method of claim 1 wherein loading the first set ofdata into the database and loading the second set of data into thedatabase includes using a lidar to obtain the first set of data and thesecond set of data.
 7. The method of claim 1 wherein loading the firstset of data into the database includes using a first lidar to obtain thefirst set of data and loading the second set of data into the databaseincludes using a second lidar to obtain the second set of data.
 8. Themethod of claim 7 wherein using the first lidar to obtain the first setof data includes positioning the first lidar at the first vantage pointand using the second lidar to obtain the second set of data includespositioning the second lidar at the second vantage point.
 9. The methodof claim 8 wherein using the first lidar to obtain the first set of datais done simultaneously with using the second lidar to obtain the secondset of data.
 10. The method of claim 1 wherein determining a globalposition of the objects in the area based on information in the databaseincludes monitoring movement of the objects within the area.
 11. A videosurveillance system comprising: a lidar; a global positioning system;and a processor that receives data from the lidar relating to the sizeand distance of objects in an area from a first vantage point and datarelating to the size and distance of objects in the area from a secondvantage point, the processor further receiving data from the globalpositioning system relating to global positions of the first vantagepoint and the second vantage point and based on the data creates asimulated image of the area from an orientation above the area whichindentifies locations within the area that may not be seen by videosurveillance from the first vantage point and the second vantage point.12. The video surveillance system of claim 11 further comprising a videocamera, wherein the video camera delivers surveillance video to theprocessor from the first vantage point and the second vantage point andbased on the surveillance video the processor creates a simulated imageof the area from an orientation above the area which indentifieslocations within the area that may not be seen by the video camera fromthe first vantage point and the second vantage point.
 13. The videosurveillance system of claim 12 wherein the video camera is a firstvideo camera located at the first vantage point, and the videosurveillance system further comprises a second video camera located atthe second vantage point.
 14. The video surveillance system of claim 13wherein the first video camera and the second video camerasimultaneously deliver surveillance video to the processor.
 15. Thevideo surveillance system of claim 11 wherein the lidar is a first lidarthat delivers data to the processor relating to the size and distance ofobjects in the area from the first vantage point, and the videosurveillance system further comprises a second lidar that delivers datato the processor relating to the size and distance of objects in thearea from the second vantage point.
 16. The video surveillance system ofclaim 15 wherein the first lidar and the second lidar simultaneouslydeliver data to the processor.
 17. The video surveillance system ofclaim 15 wherein the first lidar, the second lidar and the globalpositioning system simultaneously deliver data to the processor.
 18. Thevideo surveillance system of claim 17 wherein the first lidar and thesecond lidar monitor movement of the objects within the area.
 19. Avideo surveillance system comprising: a first lidar located at a firstvantage point; a second lidar located at a second vantage point; a firstvideo camera that delivers surveillance video from the first vantagepoint; a second video camera that delivers surveillance video from thesecond vantage point; a global positioning system; and a processor thatreceives data from the first lidar and the second lidar relating to thesize and distance of objects in an area from the first vantage point anddata relating to the size and distance of objects in the area from thesecond vantage point, the processor further receiving surveillance videofrom the first camera and the second camera, the processor furtherreceiving data from the global positioning system relating to globalpositions of the first vantage point and the second vantage point andbased on the data creates a simulated image of the area from anorientation above the area which indentifies locations within the areathat may not be seen by first video camera and the second video camera.20. The video surveillance system of claim 19 wherein the first lidarand the second lidar monitor movement of the objects within the area.